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ABSTRACT 


Educators are increasingly embracing personalization in online 
and blended learning programs as a means of focusing students’ 
investment of time and energy into learning plans that are best 
tailored to their individual needs. When personalized learning 
tools are deployed into structured learning environments like 
schools, however, educators and students must consider program 
provided recommendations alongside potentially immutable 
factors like set daily schedules, mandated curricula, and student 
needs in other content areas. These on-the-ground factors make 
researching the impacts of personalized learning challenging 
because they are difficult to measure directly, especially for digital 
programs deployed at scale. Inspired by a widely influential 
methodology in brain imaging, we tackled this challenge by 
employing an event-related approach that emphasizes changes in 
student behavior that are time-locked to changes in program 
provided usage recommendations. Our analysis reveals that while 
student usage time can often be quite far from the amount 
recommended, students nevertheless respond to changes in 
program recommendations by adjusting usage in a corresponding 
manner. We further extend this general approach to demonstrate 
that students more often stayed on track toward their end of year 
goals following a week where they met or exceeded their program 
provided recommendation. Through these examples, we 
demonstrate the value of an event-related approach towards 
understanding how personalized paths can positively influence 
student learning. 
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1. INTRODUCTION 


As schools and communities embrace a rapidly changing world, a 
growing emphasis on the personalization of learning has emerged 
[10]. Learning is considered personalized if it is tailored to each 
learner’s strengths, needs, and interests, encouraging flexibility in 
a student’s pursuit of mastery and enabling learners to take an 
active role in what, when, where, and how they learn [22]. In the 
competition for instructional time, personalized learning 
approaches also hold the promise of helping students achieve 
mastery as efficiently as possible [10], and can facilitate 
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educators’ work in guiding students’ learning efforts towards 
educational activities that best match their current needs. 


Online and blended learning programs are uniquely positioned to 
enable personalized learning because they can support student 
agency through independent pacing, delivery of differentiated 
content and support, and the ability to engage with learning 
anytime and anywhere [22]. However, the double-edged sword of 
personalized learning is that “the process of personalization puts 
enormous pedagogical and procedural burden on the students—as 
well as teachers—to make critical instructional decisions” [4; also 
see 5]. This includes decisions about how much time students 
should spend on specific programs and components of programs 
to maximize learning. While studies often find that students fail to 
spend as much time in educational technology programs as 
recommended by the program or researchers [23], students can 
also over-use, spending time on one set of activities that might be 
better spent in other areas. 


One response from the designers of learning technologies has 
been the inclusion of embedded recommendations and self- 
monitoring tools to scaffold student and teacher support for self- 
regulation. Recommendations are tailored to help students and 
teachers make good decisions within a personalized learning 
environment without enforcing rigid requirements that may reduce 
student agency and be unrealistic for particular educational 
contexts. Individualized usage time recommendations do not 
appear to be common in most learning technologies; many 
continue to provide one-size-fits-all usage recommendations [9]. 
However, they hold the promise of facilitating self-pacing by 
helping students who are at different levels and progressing at 
different speeds to stay on track toward reaching their goals. 


Despite the recognition of learning scaffolds as critical and 
effective for self-regulation in general [15] and in computer-based 
learning environments in particular [27, 28], relatively little 
research has been done into the impacts of recommendations. 
While the desire to enact personalization grows, the reality is that 
many educational institutions, particularly K-12 schools, continue 
to look much as they have for the past century, with set daily 
schedules and highly-regulated or mandated paths through content 
material [10]. When individualized learning tools are deployed 
into schools with structured learning environments, educators and 
students must consider program provided recommendations 
alongside these potentially immutable factors. While a program 
may recommend a different usage time to individual students 
within the same class or to the same student in different months, 
they may be unable to follow those recommendations with fidelity 
because of set schedules of technology access [25], challenges 
associated with implementing flexible learning time [20], or 
teacher and parent beliefs about learning technologies and screen 
time [6, 18]. Furthermore, researchers often have data on the 
usage recommendations a student received and their time spent 
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using the program, but lack direct insight into the specific 
contexts in which the program is implemented [26]. 


We addressed this research challenge by examining how students 
respond to personalized usage time recommendations within 
Lexia® CoreS°® Reading (hereafter “Core5’”) - a blended learning 
literacy program that provides instruction in foundational literacy 
skills for students in grades preK-5. To isolate the impacts of 
program provided recommendations from the largely unknown 
aspects of school context, we model an analytic approach after 
one widely used in the field of brain imaging - event-related 
design [for a general overview see 12, 13; for a widely-cited early 
example see 11]. The key aspect of this methodology with respect 
to our present application is that we focus on changes in actual 
student usage time that occur time-locked to changes in 
personalized recommendations. That is, we ask how student 
behavior responds to changes in program suggestions rather than 
whether it is aligned to recommendations, as consistent student 
responses to changed program recommendations could be 
observed even if baseline student usage is widely variable across 
diverse school contexts. Utilizing this approach, we find that 
students indeed adjust the amount of time per week that they 
spend using Core5 in a manner that differentially relates to the 
direction and magnitude of the recommended change. We also 
extend our analysis to examine events defined directly by student 
behavior and find that the act of meeting one’s recommendation 
in a given week is associated with more frequently staying on 
track toward end-of-year goals in future weeks. Together, these 
examples highlight the power of an event-related approach, and 
reveal positive associations between the personalization of 
learning and student progress within school contexts. 


2. DATA 


2.1 Usage time recommendations in Core5 

To personalize the learning path for each individual student, 
Core5 recommends a number of minutes per week that the student 
should use the online portion of the program, promoting regular, 
right-sized use and proactive time management throughout the 
school year to enable student success [7]. Each student’s usage 
recommendation reflects the estimated amount of time needed to 
reach their end of year “benchmark” - that is, to complete all 
program content for their grade level by the end of the school 
year. It is based on a predictive model that takes into account a 
student’s current place in the program, the amount of material left 
for them to reach their benchmark, and their time spent and 
progress made in the prior month [16]. These recommendations 
are shared prominently with educators in the program’s online 
data portal, and are visible to students while logged in to CoreS5. 


Critically, student usage recommendations are not fixed 
throughout the school year, but are recalculated at the start of each 
month to reflect student progress and pace (see Figure 1). At the 
start of the year, before enough data has been collected to 
personalize recommendations, all students are set to a default 
recommendation of 40 minutes per week. At the start of the next 
calendar month (first Monday), a student’s recommendation 
changes to 20, 30, 50, or 60 minutes per week. With the 
beginning of each new month, a student’s usage recommendation 
is recalculated, resulting in either an additional change or a static 
recommendation. This cadence was chosen to allow regular 
revisions that reflect student’s usage and progress, while still 
remaining implementable for teachers. The goal in personalizing 
and updating these recommendations is that students use the 


program enough to stay on pace to end the year at their grade 
level benchmark, without spending more time than necessary that 
could be invested in other learning activities. Previous research 
has shown that students who consistently meet recommended 
usage in Core5 make more progress and more often reach their 
grade-level benchmark than those students who infrequently or 
never meet their usage recommendation [17]. 


ese Student 1 
mee Student 2 


recommended usage, minutes 


Aug. 6 
Sept. 3 
Oct. 1 
Nov. 5 
Dec. 3 


Figure 1: Usage recommendation profiles for two example 
students. Each line illustrates how usage recommendations for 
an individual student change across the time frame under 
study. Student 1 began the school year in early August, and 
like all students was initially recommended 40 minutes per 
week. On the first Monday of September, Student 1’s 
recommendation changed to 50 minutes per week, and at the 
start of October it was adjusted again to 60 minutes per week. 
Student 2 was also initially set at 40 minutes per week, but 
began the school year later (in mid-September). Following the 
same rules, however, Student 2’s recommendation was 
adjusted at the start of the next calendar month (October) to 
30 minutes per week. For this student, the recommendation 
remained there for the duration of this time frame. 


2.2 Sample Details 

Weekly usage records for Core5 students in Kindergarten through 
3" grade were used for the analyses presented in this paper. 
Although Core5 also provides usage recommendations for pre-K, 
4th, and 5th grade students, the specific time values differ for 
these grade levels. We therefore restricted our sample to K-3 
students for clarity of interpretation, though we anticipate that 
results would be similar for students in other grades. 


To obtain the records, schools were chosen at random from 
among those who had at least one student using Core5 in the fall 
of 2018 (total of 168 schools chosen). These schools were 
geographically diverse, located across 39 US states and 4 
Canadian Provinces. Student-level demographic data is 
unavailable for this dataset. All weekly Core5 usage records 
between August 6, 2018 and December 31, 2018 were obtained 
for all students at these schools. To be included in the final 
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sample, students must have used Core5 for at least 7 weeks within 
this date range, and had their usage recommendation change at 
least once (as described in Section 1, the goal of the event-related 
approach is to focus on these changes). In addition, students who 
met their end of year benchmark (completed all grade level 
material) within this timeframe were excluded (this sets ones’ 
usage recommendation to 0 minutes per week). Furthermore, 
students must have had a usage target of 40 minutes in their first 
week of program use within this timeframe. As previously 
described, Core5 assigns a default recommendation of 40 minutes 
per week during a student’s first month of use in a school year, 
and any other value at that time point is an indication that there 
was a manual override (this is rare - 0.9% of students in our 
sample - but an available option for educators). The final sample 
contained 10,851 students (2,838 in Kindergarten; 3,213 in 1* 
grade; 2,836 in 2™ grade; 1,964 in 3" grade). To ensure that these 
exclusion criteria did not produce non-representative results, we 
ran robustness checks using different cutoffs for minimum weeks 
of program use (6 or 8) and repeated our analyses with two 
additional samples of students based on new random selections of 
schools. We found that all results were qualitatively consistent 
with our reported findings. 


The weekly Core5 records obtained contain aggregated usage data 
for each week that a student logged into the program. The metrics 
collected that are relevant to the presented analyses include the 
total time of Core5 use during that week, the recommended use 
time for that week, whether or not a student met their 
recommendation (total time greater than or equal to recommended 
time), and the Monday date of the week reported. 


3. RESULTS 


3.1 Alignment to usage recommendations 

Our primary research aim is to assess whether students’ usage 
time is responsive to Core5’s personalized recommendations. 
Before presenting our results, however, it is critical that we 
distinguish this question of students’ responsiveness to 
personalized recommendations from a related question about 
alignment between recommended and actual usage time. 
Specifically, we could observe changes in actual Core5 usage 
following a change in the program provided recommendation (i.e. 
a personalized response) without necessarily finding that students 
used the program for a particular number of minutes that is close 
to their recommended value (i.e. alignment). Indeed, because 
Core5’s personalized recommendations serve as only one factor 
within the school context, it would not be surprising if a student’s 
usage time in a given week was quite far from their personalized 
recommendation value, and more closely related to unknown 
(from a researcher’s perspective) contextual factors such as the 
amount of time dedicated in their school’s schedule to literacy 
learning or student-directed after school usage. Critically, even if 
there is poor alignment, we may find that when recommendations 
are changed that students’ time spent using the program 
systematically adjusts in a manner consistent with those changes - 
a result indicative of responsiveness to Core5 recommendations. 


We indeed find that alignment between Core5’s usage 
recommendations and actual student usage time is weak. Although 
most students had a mean weekly usage time that fell within the 
range of Core5 recommendations (Figure 2, top panel; 70.0% of 
students with weekly mean between 20 and 60 minutes), there was 
a small negative correlation between actual and recommended 
program use time in aggregate (Pearson r = -0.117; 95% CIs = - 


0.137, -0.098). Honing in on a snapshot of one particular week in 
our dataset (Figure 2, bottom panel), it is evident both that there is 
poor alignment to recommendations, and that there is widespread 
individual variability in usage time. While the average within- 
student mean for actual and recommended usage time were similar 
(46.2 and 43.3 minutes per week, respectively) the across-student 
standard deviations for these metrics were widely disparate (SDs 
= 22.9 and 11.1 minutes, respectively). 
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Figure 2: Distributions of students’ Core5 usage time. (top) 
Though mean weekly usage across students was in a range 
similar to Core5’s usage recommendations, there was a 
notable extent of individual variability. (bottom) A snapshot of 
the distribution of actual use (y-axis) for students with each 
unique recommended usage time during a particular week (x- 
axis) revealed no apparent relationship between the two. The 
data shown is for one example week that had the largest 
number of unique students using Core5 (week beginning 
November 26, 2018; 9,606 students, or 88.5% of full sample, 
had Core5 use), but other weeks had qualitatively similar 
relationships. The correlation between recommended and 
actual usage was similar for this sample week (Pearson r = - 
0.043, 95% CIs = -0.063, -0.023) to that seen for the aggregate 
results. 
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3.2 Event-related approach to 


recommendation changes 

In light of these observations indicating a lack of positive 
alignment (Figure 2), we turned to our key question of whether 
student usage time in Core5 was nevertheless responsive to 
changes in personalized recommendations. Because of the 
complex and largely unknown (from a researcher’s perspective) 
context in which these personalized recommendations are 
implemented, we turned to an event-related analytic approach. 
This methodology allows for context-independent examinations of 
an event of interest by effectively contrasting responses that occur 
in temporal coordination with that “event” against a baseline 
period just before the event occurred [11]. 
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Figure 3: Event-related analysis of usage recommendation 
changes. When a student’s weekly usage recommendation 
changes from 40 minutes (Time -1) to another duration (Time 
0-2) early in a new school year, student usage tends to change 
in the direction of (and with magnitude correlated to) the 
change in recommendation. Each bold line represents the 
median change in actual weekly usage time across students, 
with all usage times in the analysis expressed as a difference 
from Time -1 (this is why all lines converge to 0 at Time -1). 
Shaded areas around each line represent 95% confidence 
intervals on the median generated via a bootstrap resampling 
procedure. 


The first step in conducting this analysis is to define the event of 
interest - here, we focused on how each student’s first change in 
recommendation influenced their usage time within Core5. Note 
that because all students began the school year with a 
recommendation of 40 minutes, this event reflects a change from 
40 minutes (at Time -1 in Figure 3) to one of the other four 


possible usage recommendations (20, 30, 50, or 60 minutes at 
Time 0-2 in Figure 3). We next aligned all student data to a 
temporal reference frame defined by this event. In other words for 
each student, we defined Time -1 as their last week of program 
use prior to the recommendation change, Time 0 as the week 
when the new recommendation first appeared, and Times 1 and 2 
as the next two weeks during which that same recommendation 
remained. Note that these weeks are ordered but are not 
necessarily consecutive, as students do not always use the 
program every week. This means it was possible for a student’s 
recommendation to change again at Time 1 or 2 if it fell in the 
next calendar month. To ensure that the time-course analyzed in 
Figure 3 reflects the response to the initial target change, we 
excluded 363 students (3.3%) for whom this occurred, leaving a 
sample of 10,488. Finally, we subtracted out each student’s actual 
usage time at Time -1 from all 4 time points to yield a difference 
metric (this is why all lines converge at 0 for Time -1). 


Figure 3 shows the median event-related change in actual student 
usage time when a recommendation changes from 40 minutes per 
week at the start of the school year to another value. A two-factor 
ANOVA (factors of recommendation and time from 0 to 2 in 
Figure 3, the latter as a repeated measure) revealed significant 
main effects of both recommendation and time (F, jo.=93.564, p < 
0.0001, partial n*= 0.0260; F, 1; = 5.71, p < 0.0001, partial n= 
0.0005, respectively), as well as a significant interaction between 
time and recommendation (F, »»; = 10.10, p < 0.0001, partial 1?= 
0.0030). Repeating this statistical test with a sample that excluded 
outliers (353 students, or 3.4%, with a change at any time point 
more than 3 SDs from the mean) produced the same pattern of 
results. These findings clearly indicate differential responses to 
Core5 usage recommendations, with decreases in recommended 
usage (from 40 to 20 or 30 minutes per week) tending to result in 
decreases in program use, and increases in recommended usage 
(from 40 to 50 or 60 minutes per week) tending to result in 
increases. Interestingly, the response to a recommendation change 
appears to unfold in time, with students continuing to adjust usage 
time in the direction that their recommendation changed over the 
next few weeks. This finding further emphasizes the limitations of 
using snapshot analyses like those in Figure 2 to tease apart 
effects with unknown temporal dynamics. 


It is notable that the median change in program use was smaller in 
magnitude than the recommended change, especially when one’s 
Core5 recommendation increased. This observation is consistent 
with the explanation that contextual factors specific to each 
student’s school and situation are weighed alongside the 
program’s personalized recommendations. We also found that 
despite the visible responses to recommendation changes (Figure 
3), that average usage time for all recommendation categories 
tended to hover around 40 minutes per week (e.g. means in Figure 
2, bottom). Such a result suggests a continued reliance on the 
initially recommended value of 40 minutes per week for all 
students (see Section 4). 


3.3 Event-related approach to student fidelity 


of program use 

As we have demonstrated, taking an event-related approach to 
studying learning paths in CoreS5 can clearly reveal differentiated 
student responses to personalization. While the analysis illustrated 
in Figure 3 represents one application of this approach to events 
defined directly by program-driven occurrences (changes made by 
Core5 at specific points in time), a key advantage of event-related 
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designs is their flexibility to define new events based on the 
nature of student actions as well [c.f. 8, 13]. To exemplify this 
type of approach and to gain more insight into how personalized 
recommendations influence students’ program use, we next define 
new events based on whether or not a student met or exceeded 
their recommended usage time in a given week. 


Using this definition, we can now ask whether the “event” of a 
student meeting or exceeding Core5’s usage recommendation in a 
given week is associated with a lasting impact on a student’s 
fidelity of program use, relative to weeks when that same student 
did not meet her Core5 usage recommendation. In other words, 
are these helpful recommendations that encourage students to set 
achievable targets, appropriately pace themselves, and use with 
fidelity throughout the year [19]? Because it is critical that this 
analysis be conducted in a within-student fashion (i.e. comparing 
how the same student responds to both event types), we included 
only students who had at least one instance of both meeting and 
not meeting their usage recommendation within the timeframe 
under study (N=8,911; 82.1% of full sample). 


Results indicated that students more often met or exceeded their 
Core5 usage recommendation if they had also met or exceeded 
their recommendation during their prior week of program use 
(56.0%, vs. 50.5% when they did not meet or exceed their 
recommendation during the prior week; odds ratio = 1.248). We 
also found that while it was very likely overall for students to use 
Core5 in consecutive weeks, that this was even more frequent 
following a week of meeting than not meeting one’s usage 
recommendation (88.0% vs. 83.6%, odds ratio = 1.438). Together, 
these results suggest that following personalized usage 
recommendations is associated with staying on track toward end 
of year goals and maintaining regular program use. 


4. DISCUSSION 


While measuring the impacts of personalized learning in school 
settings carries significant challenges, we demonstrate the power 
of an event-related analytic approach toward revealing how 
student behavior responds to program provided recommendations. 
Clearly, educators and students must make decisions about 
personalized recommendations within the context of their school 
environment and alongside myriad other considerations. The 
apparent lack of alignment between actual program use and Core5 
recommendations (Figure 2), then, is a manifestation of these 
important but competing priorities. Using an event-related design, 
we were able to reveal that even within this complex ecosystem, 
students’ Core5 usage time does change in a manner that directly 
corresponds (and is time-locked) to changes in their personalized 
recommendations. Furthermore, our results demonstrated that 
students more often stayed on track toward their end-of-year 
target following weeks in which they met, versus lagged behind, 
their suggested pace. 


Although usage recommendations are visible to students in the 
Core5 program, given our sample’s age group (K-3) we expect 
that teachers and school administrators are primarily responsible 
for monitoring Core5 usage time, responding to 
recommendations, and weighing program time against other 
educational priorities. This balancing act likely explains why 
students’ usage time adjustments were typically smaller than was 
recommended (Figure 3). Together with our other findings, this 
pattern is consistent with program provided recommendations 
influencing but not determining student usage time when they are 


considered alongside additional factors in each unique school 
context. In future work it will be interesting to investigate whether 
responsivity and/or alignment to usage recommendations changes 
with student age, perhaps reflecting increasing self-regulation and 
autonomy as they advance in school. 


While the ability to isolate one factor of interest from within a 
complex, dynamical system is a key strength of an event-related 
approach, it is also a limitation in that it does not afford the ability 
to quantify influences of other factors or to provide insight as to 
their relative importance. From the perspective of those designing 
and improving personalized learning tools, however, an event- 
related approach is powerful for exactly that reason - it allows for 
isolated study of a personalized feature that is directly within the 
designer’s control, thus facilitating improvement of the program’s 
design and iteration on these enhancements [c.f. 14]. For example, 
we noted an interesting finding that even as student usage times 
changed in response to program provided recommendations, they 
seemed to remain tied to the initial, impersonal value of 40 
minutes per week (e.g. Figure 2, bottom). This pattern may reflect 
a well-studied cognitive bias known as anchoring, which typically 
manifests as a continued reliance on an initially given value when 
making numerical judgments [24]. It may also be that educators 
have more flexibility to adjust student schedules early in the year 
than they do as school progresses. In either event, this result 
suggests that personalizing a student’s usage recommendation 
earlier in the school year could yield larger impacts. 


By extending our event-related approach, we found that weeks in 
which a_ student met or exceeded their personalized 
recommendation were more often followed by continued on-track 
behavior and more regular program use, which have previously 
been shown to be positive predictors of student performance in 
online courses [c.f. 7, 21]. Such an effect may stem from 
integration of Core5’s personalized recommendations within 
educators’ learning plans and/or with students’ emerging self- 
regulation [1, 2, 19]. As previously described, our event-related 
approach limits our ability to quantify effects beyond those owing 
to program provided recommendations by intentionally filtering 
them out to isolate only a single factor. That said, these findings 
motivate further study of the mechanisms through which usage 
recommendations facilitate students’ ability to stay on track for 
success throughout the school year. 


We also note that while our approach is inspired by one 
developed for the analysis of brain imaging data, it differs in 
important ways. First and foremost, event-related designs in brain 
imaging research are typically used in the context of randomized 
studies, where an experimenter controls many aspects of the 
timing and context of “events” (although note that the ability to 
flexibly define events post-hoc is a key methodological advantage, 
c.f. 8). In contrast, Core5 students are assigned usage 
recommendations based on their pace through content material 
and the amount they have left to finish that year. By definition, 
then, students who are farther behind in class will tend to receive 
higher Core5 usage recommendations. Although the analyses we 
present highlight within-student usage changes in response to 
time-locked events, it is important to note that the groups of 
students at each recommendation level (e.g. at Times 0-2 in 
Figure 3) likely differ in other key ways. For example, we may 
speculate that one reason why usage time increases were typically 
of smaller magnitude than usage time decreases (Figure 3) could 
be that students who are farther behind tend to receive offline 
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interventions at school rather than additional time in the online 
program. 


Analytic applications in the field of brain imaging also suggest 
extensions of this work that could yield continued insights into 
the impact of personalization in learning. For example, once well 
characterized, event-related time courses serve as a template for 
identifying structural brain regions with particular functional 
properties [8, 13]. Analogously, having defined the typical time 
course of how student usage responds to recommendation changes 
(Figure 3), we could now use these expected functions as 
regressors to identify schools where recommendations are or are 
not strongly implemented. This in turn could help guide vendors 
to better help schools resolve issues and successfully implement 
digital learning tools. It could also motivate additional research 
studies that compare student outcomes in school contexts where 
personalized recommendations either were or were not 
implemented with fidelity. Such investigations will yield a deeper 
understanding of the value of personalized recommendations 
within schools, and in turn provide examples that enable 
educators to operationalize personalization in their classrooms. 
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